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LONG  TERM  GOALS 

The  development  of  a  three-dimensional,  high  resolution  nowcast/forecast  system  for  the  Arabian  Gulf 
and  Gulf  of  Oman  which  encompasses  scales  of  10  km  or  less  when  warranted,  using  the  most 
advanced  finite  element  (FE)  coastal  hydrodynamic  models.  The  modeled  dynamics  are  to  include  tidal 
and  wind-driven  flow,  buoyancy  forcing,  surface  heat  flux,  river  inflow,  and  turbulent  mixing 
processes. 

OBJECTIVE 

The  primary  objective  is  development  of  a  circulation  model  for  the  Arabian  Gulf  and  connecting 
waters  that  realistically  predicts  the  complex,  3-D  circulation  and  mixing  patterns  in  the  region  over 
seasonal,  tidal,  sub-tidal,  and  storm  event  time  scales.  Dynamical  processes  to  be  considered  include 
the  three  dominant  external  forcings  in  the  region,  a  strong  evaporative  flux,  seasonal  wind  forcing, 
and  freshwater  river  discharge.  Not  only  are  realistic  current  fields  sought  but,  the  hope  is  to  gain 
understanding  of  the  role  each  forcing  mechanism  plays  in  the  strongly  thermohaline-driven 
circulation.  A  particular  emphasis  will  be  placed  on  the  three-dimensional  currents  and  transport  of 
mass,  salt  and  heat  through  the  Strait  of  Hormuz.  Another  aim  of  this  study  is  to  demonstrate  the  utility 
of  the  finite  element  approach  using  state-of-the-art,  physically  advanced,  3-D  numerical  models. 

APPROACH 

In  working  toward  a  nowcast/forecast  predictive  capability  in  the  Arabian  Gulf,  two  circulation  models 
are  employed.  The  first  model,  the  Dartmouth  College  QUODDY  model,  represents  the  most 
dynamically  advanced  FE  model  to  date.  QUODDY  is  a  3-D,  fully  nonlinear  model  that  includes  tidal, 
wind-driven,  and  baroclinic  physics,  incorporating  advanced  turbulence  closure.  This  model  is  applied 
over  seasonal  and  synoptic  event  time  scales  towards  then  development  of  a  realistic  circulation 
climatology  and  in  the  assessment  of  individual  forcing  contributions  to  the  overall  circulation.  A 
second  model,  ADCIRC,  is  advancing  to  include  a  baroclinic  component  (development  by  Dr.  Rick 
Luettich  at  UNC).  This  model  is  currently  exercised  operationally  at  NAVOCEANO  as  a  nonlinear 
simulator  of  tidal  and  wind-driven  circulation.  ADCIRC  is  a  more  probable  candidate  for  real-time 
predictions  and  subsequent  transition  to  operations,  in  the  context  of  Naval  applications,  though  it  has 
lagged  QUODDY  with  regard  to  the  inclusion  of  baroclinic  dynamics.  Both  finite  element  models  are 
designed  with  modular  dynamics  in  which  certain  mechanisms,  such  as  heat  flux,  wind  forcing, 
stratification,  tides,  or  river  inflow  can  be  independently  included  or  excluded  from  model  equations. 
This  modularity  is  used  to  examine  the  contributions  of  each  component  to  the  overall  circulation 
dynamics. 
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WORK  COMPLETED 


The  number  of  degrees  of  freedom  associated  with  3D,  baroclinic  models  dramatically  increase  over 
that  of  2D  applications  forcing  consideration  of  computational  cost  for  a  given  application.  One 
approach  to  this  problem  is  to  limit  mesh  size  by  using  either  smaller  spatial  extents  or  courser 
resolutions.  Another  approach  is  to  expedite  solution  of  the  model  equations  through  the  use  of  parallel 
processing  techniques.  Finite  element  models  are  well-suited  to  parallelization  due  to  their  element- 
based  computations.  The  Dartmouth  College  circulation  model,  QUODDY  is  now  parallelized  using 
Open-MP  directives  and  executes  on  shared  memory  multi-processor  machines  or  single  multi¬ 
threaded  processors  of  distributed  memory  machines.  The  Open-MP  modifications  are  implemented 
within  and  tested  for  the  latest  release  of  QUODDY,  version  5.1.  Collaborations  with  the  HPCMO 
PET  program  (Dr.  Timothy  Campbell)  have  made  this  effort  a  reality.  Only  5%  of  the  code  remains 
exclusively  serial;  such  sections  are  largely  devoted  to  I/O  operations  and  user-defined  subroutines. 

The  QUODDY  model  is  also  portable,  having  been  tested  on  the  SUN  Enterprise  10,000,  SGI  Origin 
2000,  and  the  IBM-SP3.  This  parallelization  effort  represents  a  tremendous  advance  in  modeling 
capability  and  will  have  drastic  implications  in  terms  of  the  number  and  resolution  of  experiments  that 
will  be  realistically  possible  using  this  model. 

The  strong  thermohaline-driven  circulation  in  the  Arabian  Gulf  provides  a  considerable  test  for  model 
temperature  and  salinity  transport  algorithms.  Evaporation  is  the  primary  driving  force  which  places 
great  importance  on  specification  of  the  surface  boundary  condition.  Presently  surface  temperatures  are 
nudged  back  to  initial  climatological  values  and  surface  salinity  is  allowed  to  freely  evolve.  As  shown 
in  Figure  1,  vertical  profiles  of  temperature  at  the  Strait  of  Hormuz  compare  favorably  to  observed 
values  while  salinity  is  clearly  over-mixed  throughout  the  water  column.  One  solution  may  be  to  also 
nudge  surface  salinity  to  initial  climatology.  This  approach  has  been  implemented  and  testing  is 
underway.  Ultimately  spatially  variable  evaporative  fluxes  will  be  required. 

A  data  assimilation  effort  began  with  the  application  of  the  linear  harmonic  inverse  model, 

TRUXTON,  to  the  Arabian  Gulf.  Assimilated  data  includes  shipboard  ADCP  data  acquired  from 
NAVOCEANO  and  International  Hydrographic  Office  (IHO)  tidal  station  data.  Finer  points  regarding 
the  application  of  TRUXTON  were  investigated  in  collaboration  with  Dr.  Chris  Naimie  at  Dartmouth 
College.  For  example,  a  successful  inverse  application  depends  on  appropriate  parameter  specification 
for  the  inverse  solution  technique,  interpretation  of  error  measures,  and  evaluation  of  the  inversion. 
Experiments  investigating  the  sensitivity  of  minimum  depth  and  frictional  coefficient  on  the  resulting 
inverse  solution  in  the  Arabian  Gulf  are  complete. 

RESULTS 

The  seasonal  circulation  climatology  previously  generated  by  the  state-of-the-art  QUODDY  3-D  FE 
model  for  the  Arabian  Gulf  is  used  to  test  parallelization  of  the  QUODDY  code.  Mesh  resolution  over 
the  domain  is  variable  (17440  nodes,  33322  elements),  approximately  3  km  in  depths  less  then  40  m 
and  6  km  elsewhere  out  to  the  open  ocean  boundary  at  200  m  depth  in  the  Gulf  of  Oman.  The  loop-by- 
loop  parallelization  initially  implemented  is  now  replaced  by  a  single  parallel  region  resulting  in  less 
reliance  on  OpenMP  thread  management.  Full  precision  verification  against  serial  code  computations 
is  complete  for  three  HPC  platforms:  Sun  Enterprise  10,000,  the  SGI  Origin  2000  and  the  IBM  SP3,  in 
effect  making  the  code  portable  as  well.  This  effort  has  resulted  in  near  linear  speed-up  of  model 
computations  with  respect  to  the  number  of  processors  (Figure  2a).  Speed-ups  by  factors  of  5  on  8 
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processors  to  1 1  on  32  processors  on  the  Sun  Enterprise  10,000  have  been  realized.  The  CPU  time  was 
decreased  from  9  sec/timestep  on  a  single  processor  to  less  than  2  sec/timestep  using  32  processors. 
Ultimately  a  10  day  simulation  that  had  taken  approximately  22  hours  on  one  CPU  now  runs  in  only  8 
hours  on  32  processors  of  the  shared  memory  SUN  El 0000  supercomputer.  Efficiency  is  improved  for 
cases  involving  higher  vertical  resolution  (increased  numbers  of  vertical  nodes)  as  shown  in  Figure  2b. 


JANUARY 


Figure  1.  Model-data  comparisons  between  vertical  profiles  of  salinity  (left)  and  temperature  (right) 
in  the  Strait  of  Hormuz  during  Winter  (top)  and  Summer  (bottom).  Surface  temperatures  are 
nudged  to  initial  climatological  values;  surface  salinity  is  unrestrained.  QUODDY computed 
temperatures  are  reasonably  reproduced  while  salinity  values  are  clearly  over-mixed  throughout  the 

water  column. 
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OpenMF  QUODDV;  Pecsian  Gulf  Seasonal 


QpenWP  QUO0DY;  Persian  GuF  Seasonal 


Figure  2.  Parallel  (a)  speed-up  and  (b)  efficiency  of  the  Open-MP  implementation  in  QUODDY for 
seasonal  circulation  simulations.  Near-linear  speed-up  up  to  8  processors  for  a  1 7440  node  problem. 
Efficiency  linearly  decreases  to  60-70%  for  up  to  8 processors  with  continued  slight  decreases  using 
more  processors.  Greater  numbers  of  vertical  nodes  increase  efficiency  by  10%>.  All  simulations  are 

performed  on  the  SUN  Enterprise  10,000. 
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IMPACT/APPLICATION 


Detailed  knowledge  and  understanding  of  processes  governing  shallow  water  dynamics  in  the  Arabian 
Gulf  and  Strait  of  Hormuz  address  Naval  needs  for  anticipating  variability  coastal  waters  with  respect 
to  circulation  and  water  properties  over  space  and  time  scales  relevant  to  mine-countermine, 
amphibious,  or  special  operations  in  this  priority  area.  High  horizontal  resolution  currents  assist  in  the 
planning  of  instrumentation  and  tactics  associated  with  amphibious  operations  as  well  as  search  and 
rescue  efforts.  The  Arabian  Gulf  itself  is  a  high  priority  Naval  region. 

Advantages  of  an  unstructured  grid  discretization  are  evident  in  the  placement  of  open  ocean 
boundaries,  localized  resolution  refinement,  and  representation  of  bathymetric  and  shoreline 
complexities.  A  study  of  this  scope,  encompassing  the  Arabian  Gulf,  Strait  of  Hormuz,  and  Gulf  of 
Oman  containing  localized  mesh  refinements  of  less  than  5  km,  is  unprecedented. 

TRANSITIONS 

The  transitioned  operational  FE  model  ADCIRC  successfully  completed  an  operational  test  (OPTEST) 
at  NAVOCEANO  for  the  prediction  of  tide  and  wind-forced  elevations.  Model-data  comparisons  using 
NOAA  observed  water  levels  at  10  stations  along  the  eastern  U.  S.  formed  the  basis  of  the  evaluation. 
Computed  elevations  have  a  mean  error  of  1 1-12  cm  and  a  phase  lag  of  1 1-13  minutes,  differences  that 
fall  far  below  the  OPTEST  criterion. 
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Strong  interactions  exist  with  C.  E.  Naimie  and  D.  R.  Lynch  (Dartmouth  College)  regarding  model 
development  and  application  of  data  assimilation  techniques.  Collaborations  with  R.  A.  Luettich  (U. 
North  Carolina)  are  leading  to  advancement  of  the  ADCIRC  model  to  include  baroclinic  physics. 

PUBLICATIONS 

Blain,  C.  A.,  R.  H.  Preller,  and  A.  P.  Rivera,  Tidal  Prediction  Using  the  Advanced  Circulation  Model 
(ADCIRC)  and  a  Relocatable  PC-based  System,  Oceanography,  in  review,  2001. 

Campbell,  T.  J.  and  C.  A.  Blain,  OpenMP  Parallelization  of  a  3-D  Finite  Element  Circulation  Model, 
Navigator,  NAVOCEANO  MSRC,  Spring  2001. 

Campbell,  T.  J.  and  C.  A.  Blain,  2001.  OpenMP  Parallel  Implementation  of  the  QUODDY  3-D  Finite 
Element  Circulation  Model,  NRL  Formal  Report,  NRL/FR/7320-01-xxxx,  Naval  Research  Laboratory, 
Department  of  the  Navy,  to  be  submitted  October,  2001. 

Smedstad,  Lucy  F.  and  C.  A.  Blain,  2000.  A  Review  ofU.S.  Navy  Atmospheric  Products  in  the 
Arabian  Gulf:  An  Examination  of  NORAPS  and  COAMPS,  NRL  Formal  Report,  NRL/FR/7320-01- 
9978,  Naval  Research  Laboratory,  Department  of  the  Navy,  in  review. 

Edwards,  Catherine  R.  and  C.  A.  Blain,  200 1 .  A  Evaluation  of  Tidal  Modeling  Strategy  in  the  Yellow 
Sea:  Large  Domains  versus  Limited  Area  Models,  NRL  Formal  Report,  NRL/FR/7320-0 1-9976,  Naval 
Research  Laboratory,  Department  of  the  Navy. 


5 


Edwards,  Catherine  R.  and  C.  A.  Blain,  2001.  Operational  Evaluation  of  the  ADCIRC-2DDI  Model 
for  the  Western  North  Atlantic  Ocean,  NRL  Formal  Report,  NRL/FR/7322-01-xxxx,  Naval  Research 
Laboratory,  Department  of  the  Navy,  in  review. 

Blain,  C.  A.  and  K.  Kelly,  2001.  Software  Design  Document  for  the  Advanced  CIRCulation  Model  for 
Shelves,  Coastal  Seas  and  Estuaries  (ADCIRC),  NRL  Memorandum  Report,  NRL/MR/7320-01-xxxx, 
Naval  Research  Laboratory,  Department  of  the  Navy,  in  review. 


6 


